Virtual reality (also referred to as VR) and augmented reality (also referred to as AR) systems are well known in the art. In virtual reality, an entire environment is generated by a computer, and is immersive in nature. Subject to system design considerations, the user can move freely around the environment, which may or may not reflect real world environment. The environment may reflect a model of reality, reflect a fictitious environment, or comprise a combination of the two. A common benefit of VR is that the user can view an environment from perspectives not normally possible in the real world. For example, a user can ‘fly’ above an area to get a bird's eye view.
Augmented reality combines the “real world” reality with a virtual one. The “real world” reality is the actual observable scene, one perceived by a viewer (e.g. a human eye, an optical or electronic sensor, and the like). The observable scene is determined by the viewer location, direction of view and limitations of the viewer (e.g. darkness) or the actual environment (such as fog, smoke, vegetation, and the like). In augmented reality (also referred to as AR), computer generated images are combined with an actual or a representation of the observable scene.
In both AR and VR, a viewpoint determines the viewable scene. The viewpoint is determined by the viewer coordinates, the direction of view, and the horizontal and vertical field of view. In VR systems the viewpoint is commonly defined by the user, while in AR systems, the viewpoint is determined by the observer, or viewer viewpoint, i.e. the x, y and z, coordinates and the heading, pitch and roll. In both AR and VR systems a computer renders the environment based on an environment model that may contain information required to render the computer generated graphics. Such rendition may be a photo-realistic rendition of objects, a cartographical rendition, navigational data, and the like. In some cases, the most effective rendition comprises icons, textual data, pointers, and the like.
Computerized environment models often comprise a combination of computer generated graphics with actual photographic data. Depending on the purpose of the system dynamic objects may be added by information provided by sensors like radar, sonar, magnetic, heat and other sensors that reflect a dynamically changing reality.
In VR systems the user selected viewpoint determines a portion of the computerized environment model that is rendered and presented to the user. In AR systems, the viewer may be an image sensor such as a camera, a human eye, an optical sensor such as a periscope, and the like. At a given point in time, the viewer has a viewpoint, which determine the observable scene. Commonly in an AR system, an orientation sensor is provided to sense the direction of view, and in most mobile systems, a location sensor is also required, to correlate the augmented portion of the view with the observable scene. The output of the location and orientation sensors dictates the viewpoint. The AR system generates a rendered addition, also known as an overlay, which is a rendition of a portion of the computerized environment model, defined by the viewpoint. The overlay is merged with the observable scene image, in close registration thereto, to augment the visual information supplied to the user.
An example of augmented reality system is presented in U.S. Pat. No. 6,208,933 to Lazar, directed to overlaying cartographic data on sensor based video. In this system cartographic data from a storage device is superimposed on data received from a video sensor. A location sensor is utilized to correlate the video image and the cartographic image.
VR systems are often used in simulations where a ‘global’ view is required, or for immersive situations such as game playing, training, and the like. VR systems are often used for navigation and educational purposes. An example of educational AR system is described in U.S. Pat. No. 6,175,343 to Mitchell et al. directed to a system that allows article viewing while overlaying informational or historic data overlay, to enhance the exhibit experience. A good example of navigational AR system may be found in U.S. Pat. No. 6,181,302 to Lynde, which discloses a marine navigation binoculars with virtual display superimposing real world image. The Lynde device uses orientation and positioning sensors, and overlays navigational and other data from several sources on the real world image.
AR and VR systems both utilize displays to display the computer rendered graphics. AR systems are divided to video-through and see-through displays. In see-through displays, the user sees actual light reflected from the observable scene, and the computer generated graphics are merged by optical combiners. Optical combiners commonly comprise a beam splitter inserted in the field of view of the user. However newer methods of displays include reflecting the computer data directly onto the user retina, or projecting the data in front of the user. In video through display a camera captures the observable scene and the computer generated graphics are merged with the video data. In VR systems the observable scene is not desirable, as it will confuse the view. Displays may be static such as a television or LCD, or portable, such as a heads up display, head mounted displays, handheld displays and the like.
VR systems sometimes relate the rendering directly to the observable viewpoint, but provide an image that is wholly computer generated. An example of such system is provided in U.S. Pat. No. 5,566,073 to Margolin. In this system the viewpoint is determined by the location and orientation of an airplane and a three dimensional (3D) view of the terrain is displayed in the cockpit, regardless of the visibility conditions.
Since the viewpoint of an AR system depends on the observed viewpoint that is directly related to the viewer location, certain parts of the AR data are sometimes difficult to distinguish due to cluttering. If the model dictates that many objects should be rendered those objects may overlap in a narrow field of interest, such as the horizon. VR systems allow viewing from different angles to decrease the confusion, and thus may provide a better understanding of the environment, however a view of reality captured by a human eye is often required beside computer generated graphics.
There is therefore a clear need for and a significant advantage provided by the seamless integration of AR and VR systems